A First Look at Reproducibility and Non-Determinism in CMS Software and ROOT Data

Reproducibility is an essential component of the scientific process. Including software and data with a published paper is a good step towards reproducible research. However, the presence of non-determinism in a scientific workflow can make validating results very difficult even between two runs on the same machine, the same day, and using the exact same command and parameters. But for reproducibility we should be able to validate results even when the environment changes, which is even more challenging. We explore three high level methods for dealing with non-determinism in general: 1) Domain specific methods; 2) Domain specific comparisons; and 3) Virtualization adjustments. Using a complex high energy physics workflow, we use these methods to prevent, detect, and eliminate sources of nondeterminism. We observe improved determinism using predetermined random seeds, hierarchical data comparisons, and predictable progression of system timestamps. Unfortunately, sources of non-determinism continue to exist despite the combination of all three methods. We conclude that there is room for improvement in all three methods, and identify directions that can be taken in each method to continue progress towards reproducibility.